The Python Standard Library by Example (PATRICIA BLAIR's Library)

Author:Doug Hellmann
Language: eng
Format: epub
Publisher: Addison-Wesley
Published: 2011-03-08T16:00:00+00:00

age: 0

True : /

age: 1

True : /PyMOTW/

age: 2 rereading robots.txt

False : /admin/

age: 1

False : /downloads/PyMOTW-1.92.tar.gz

A nicer version of the long-lived application might request the modification time for the file before downloading the entire thing. On the other hand, robots.txt files are usually fairly small, so it is not that much more expensive to just retrieve the entire document again.

See Also:

robotparser (http://docs.python.org/library/robotparser.html) The standard library documentation for this module.

The Web Robots Page (www.robotstxt.org/orig.html) Description of robots.txt format.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.